Skip to content

feat: add progress callback to surface assessment progress - #235

Open
thunderstornX wants to merge 1 commit into
confident-ai:mainfrom
thunderstornX:progress-callback
Open

feat: add progress callback to surface assessment progress#235
thunderstornX wants to merge 1 commit into
confident-ai:mainfrom
thunderstornX:progress-callback

Conversation

@thunderstornX

Copy link
Copy Markdown

Summary

red_team() renders progress only to the console (via rich), so a custom UI such as Streamlit (the use case in #173) has no way to display assessment progress. This adds an optional progress callback that streams the same start / advance events to the caller.

Implementation

All progress in deepteam flows through two helpers, add_pbar and update_pbar. Rather than thread a callback through the 30+ attack modules that create progress bars, a single module-level observer is invoked from those two choke points, so every progress bar reports through it automatically.

Exposed two ways:

  • red_team(..., on_progress=fn) for the common case (scoped to the run).
  • set_progress_callback(fn) for direct RedTeamer use.

The callback receives dicts such as:

{"event": "start",   "description": "📝 Evaluating ...", "completed": 0, "total": 12}
{"event": "advance", "description": "📝 Evaluating ...", "completed": 1, "total": 12}

It is fully backward-compatible (new optional argument, defaults to None), and a raising callback is swallowed so user code can never break an assessment.

Example

from deepteam import red_team

def on_progress(event):  # drive a Streamlit progress bar
    st.progress((event["completed"] or 0) / (event["total"] or 1),
                text=event["description"])

red_team(model_callback=target, vulnerabilities=[...], attacks=[...],
         on_progress=on_progress)

Testing

  • New tests/test_core/test_progress_callback.py covering: start/advance events, context-manager scoping and restoration, headless (progress-disabled) emit, and callback-exception isolation. These run against the real rich.Progress path.
  • Verified end-to-end with a real red_team() run; the callback received the full event stream across all three phases:
start / advance   💥 Generating attacks
start / advance   ✨ Simulating attacks
start / advance   📝 Evaluating vulnerability types
  • Existing tests/test_core are unaffected, and the red_team() parameter reference in the docs now lists on_progress.

Closes #173

red_team() rendered progress only to the console via rich, so a custom UI (e.g. Streamlit, the use case in confident-ai#173) had no way to display assessment progress. This adds an optional progress callback that emits the same start/advance events to the caller.

All progress flows through add_pbar/update_pbar, so a single module-level observer fired from those two choke points covers every progress bar without touching the 30+ attack modules. Exposed as red_team(on_progress=...) and the module-level set_progress_callback(). Backward-compatible (new optional param), and a raising callback is swallowed so it can never break a run.

Closes confident-ai#173
@vercel

vercel Bot commented Jun 18, 2026

Copy link
Copy Markdown

@thunderstornX is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Access to assessment logs

1 participant